High Throughput Queue Algorithms

Advanced many-core CPU chips already have few hundreds of processing cores (e.g. 160 cores in an IBM Cyclops-64 chip) and as computer architecture progresses, more and more processing cores become available. The underlying runtime systems of such architectures need to efficiently serve hundreds of processors at the same time, requiring all basic data structures within the runtime to maintain unprecedented throughput. In this paper, the design of concurrent queues is explored to meet the runtime system and algorithm demands for hundreds of simultaneous transactions being handled in real time. We explore some of the basic standard queueing techniques, showing their weaknesses in a highly parallel environment. We then develop a simple high throughput queue. The algorithm is then extended to address all cases and to provide all the functionality commonly found in queues. We test all queueing algorithms on Cyclops-64, a many-core processor by IBM. The following are the major contributions of this paper: 1. Queueing theory is used to provide a mathematical background for the inherent throughput of queueing algorithms. 2. The Circular Buffer Queue algorithm: A very high throughput, low latency queue implementation useful for most common cases. 3. The High Throughput Queue algorithm: A high throughput, low latency queue implementation that supports all traditional operations on a queue. 4. Experimental results that show that for highly parallel systems, both the Circular Buffer Queue and the High Throughput Queue allow larger scalability and higher performance than other famous, state of the art implementations. 5. We show that the queue algorithms presented are linearizable and behave like nonblocking, wait-free algorithms for practical implementations.